Add proxy and user_agent labels to user login counter metric by vapopov · Pull Request #60239 · gravitational/teleport

vapopov · 2025-10-14T19:27:58Z

In this PR, added a user agent to the client tools request header. During login, it is passed through the proxy to the auth service. Additionally, information about the public proxy address is included so it can be recorded in metrics.

The user_login_total counter now has two labels: proxy and user_agent.

The proxy_group_id label allows us to track which regions are used by customers during login.
The user_agent_type and version label helps identify the client tools version in use, to detect any affected versions in future, and monitor the progress of client tools managed updates.

Example:

# HELP user_login_per_client Number of times there was a user login with specific client version and GroupID that routes request
# TYPE user_login_per_client counter
user_login_per_client{proxy_group_id="proxy1",user_agent_type="web",version="18.2.5"} 1
user_login_per_client{proxy_group_id="proxy1",user_agent_type="api",version="18.2.5"} 1
user_login_per_client{proxy_group_id="proxy2",user_agent_type="api",version="18.2.5"} 2

tigrato · 2025-10-14T20:37:37Z

How can we ensure that an attacker doesn't break cloud monitoring by spamming tons of agents and causing Prometheus to fail? Are we going to drop them in cloud?

rosstimothy

Agree with Tiago, this seems like it could be a very high cardinality metric. cc @evanfreed.

sclevine

I think the concern about high cardinality on the user agent is worth addressing for scalability reasons, even if authenticated users could likely already cause trouble by connecting many agent versions (which we already record).

Could we parse the user agent for Teleport major version, and only accept reasonable values?

vapopov · 2025-10-14T22:16:09Z

How can we ensure that an attacker doesn't break cloud monitoring by spamming tons of agents and causing Prometheus to fail? Are we going to drop them in cloud?

@tigrato in order to do that, you need a user from the cluster and must complete the login process. We would probably need to reset counters periodically as well

Replaced Sprintf with string concatenation

…agent-proxy-tags-metric

vapopov · 2025-10-31T01:51:54Z

I’ve added a periodic cleanup (hourly) and implemented the label separation by type and version as proposed by @evanfreed.
Since the metric is periodically reset, I’ve introduced a separate metric, user_login_per_client, to avoid breaking existing dashboards built on user_login_total.

vapopov · 2025-11-07T21:44:47Z

Any other suggestions or concerns regarding this PR? cc: @tigrato @rosstimothy @evanfreed @hugoShaka

rosstimothy · 2025-11-10T19:42:48Z

@evanfreed do you have any concerns about consuming this metric in Cloud?

Login counter metric should not be exported.

evanfreed · 2025-11-12T18:43:16Z

@evanfreed do you have any concerns about consuming this metric in Cloud?

With the recent adjustments we should be good. Appreciate keeping an eye out for that.

vapopov · 2025-11-17T21:57:04Z

@rosstimothy could you please take a look on addressed changes aa9f142

CR changes

backport-bot-workflows · 2025-11-20T10:01:05Z

@vapopov See the table below for backport results.

Branch	Result
branch/v17	Failed
branch/v18	Create PR

* Added user login counter metrics labels with proxy information and user-agent version * Fixed defining default user-agent when `extraHeaders` map was nil Replaced Sprintf with string concatenation * Fix the env proxy group ID * Add periodic clean up and change label for the metric * Use `ProxyGroupID` from the proxy configuration. Login counter metric should not be exported. * Namespace new metric CR changes * Removed global constants and changed help message * Comments adjustment

…#61622) * Added user login counter metrics labels with proxy information and user-agent version * Fixed defining default user-agent when `extraHeaders` map was nil Replaced Sprintf with string concatenation * Fix the env proxy group ID * Add periodic clean up and change label for the metric * Use `ProxyGroupID` from the proxy configuration. Login counter metric should not be exported. * Namespace new metric CR changes * Removed global constants and changed help message * Comments adjustment

Added user login counter metrics labels with proxy information and us…

bb66d5c

…er-agent version

vapopov requested review from hugoShaka, sclevine and strideynet October 14, 2025 19:27

vapopov requested review from r0mant, rosstimothy, russjones and zmb3 as code owners October 14, 2025 19:27

vapopov added the no-changelog Indicates that a PR does not require a changelog entry label Oct 14, 2025

vapopov requested a review from tigrato as a code owner October 14, 2025 19:28

vapopov added backport/branch/v16 backport/branch/v17 backport/branch/v18 labels Oct 14, 2025

github-actions Bot added the size/sm label Oct 14, 2025

github-actions Bot requested a review from bernardjkim October 14, 2025 19:28

rosstimothy reviewed Oct 14, 2025

View reviewed changes

Comment thread integrations/event-handler/go.mod

Comment thread lib/auth/sessions.go Outdated

Comment thread lib/auth/sessions.go Outdated

Comment thread lib/client/weblogin.go Outdated

sclevine reviewed Oct 14, 2025

View reviewed changes

Comment thread lib/auth/sessions.go Outdated

Fixed defining default user-agent when extraHeaders map was nil

3cc147e

Replaced Sprintf with string concatenation

vapopov force-pushed the vapopov/user-login-agent-proxy-tags-metric branch from cbec0d8 to 3cc147e Compare October 15, 2025 05:34

vapopov added 3 commits October 28, 2025 02:21

Merge remote-tracking branch 'origin/master' into vapopov/user-login-…

bc7d8f0

…agent-proxy-tags-metric

Fix the env proxy group ID

9470d9a

Add periodic clean up and change label for the metric

23a5e78

rosstimothy reviewed Nov 10, 2025

View reviewed changes

Comment thread lib/web/apiserver.go Outdated

Comment thread lib/auth/auth.go Outdated

Use ProxyGroupID from the proxy configuration.

aa9f142

Login counter metric should not be exported.

tigrato removed the backport/branch/v16 label Nov 11, 2025

tigrato approved these changes Nov 12, 2025

View reviewed changes

rosstimothy reviewed Nov 18, 2025

View reviewed changes

Comment thread lib/auth/auth.go Outdated

Comment thread lib/auth/auth.go Outdated

Comment thread lib/auth/auth.go Outdated

Comment thread lib/web/apiserver.go Outdated

rosstimothy self-requested a review November 18, 2025 14:17

Namespace new metric

bdadb18

CR changes

rosstimothy reviewed Nov 19, 2025

View reviewed changes

Comment thread lib/auth/auth.go Outdated

Comment thread lib/auth/auth.go Outdated

Comment thread lib/auth/auth.go Outdated

vapopov requested a review from rosstimothy November 19, 2025 18:27

vapopov force-pushed the vapopov/user-login-agent-proxy-tags-metric branch from c074d8f to 54a8560 Compare November 19, 2025 18:44

Removed global constants and changed help message

cb31dbc

vapopov force-pushed the vapopov/user-login-agent-proxy-tags-metric branch from 54a8560 to cb31dbc Compare November 19, 2025 18:45

rosstimothy approved these changes Nov 19, 2025

View reviewed changes

public-teleport-github-review-bot Bot removed request for bernardjkim, hugoShaka and strideynet November 19, 2025 19:04

Comments adjustment

ddf0952

vapopov requested review from hugoShaka and strideynet as code owners November 19, 2025 19:39

vapopov added this pull request to the merge queue Nov 20, 2025

Merged via the queue into master with commit e2d968e Nov 20, 2025
45 checks passed

vapopov deleted the vapopov/user-login-agent-proxy-tags-metric branch November 20, 2025 09:58

vapopov mentioned this pull request Nov 20, 2025

[v18] Add proxy and user_agent labels to user login counter metric #61602

Merged

vapopov mentioned this pull request Nov 20, 2025

[v17] Add proxy and user_agent labels to user login counter metric #61622

Merged

Uh oh!

Conversation

vapopov commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tigrato commented Oct 14, 2025

Uh oh!

rosstimothy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sclevine left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vapopov commented Oct 14, 2025

Uh oh!

vapopov commented Oct 31, 2025

Uh oh!

vapopov commented Nov 7, 2025

Uh oh!

Uh oh!

Uh oh!

rosstimothy commented Nov 10, 2025

Uh oh!

evanfreed commented Nov 12, 2025

Uh oh!

vapopov commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

backport-bot-workflows Bot commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vapopov commented Oct 14, 2025 •

edited

Loading

sclevine left a comment •

edited

Loading